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« The MAILING DATE of this communication app ars on the cov r sheet with th correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1)S Responsive to communication(s) filed on 14 November 2002 . 
2a)D This action is FINAL. 2b)|3 This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) |3 Claim(s) 1-9,11-32 and 34-44 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) 03 Claim(s) 1-9.11-32 and 34-44 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

1 1) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) Q/ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 . 
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DETAILED ACTION 

1. This action is responsive to communications: RCE filed filed 1 1/14/2002, to the original 
application filed 12/31/1998. 

2. The rejection of claims 6, 29 under 35 U.S.C. 1 12, second paragraph, has been withdrawn as 
necessitated by amendment. 

3. The rejection of claims 1-21, 23-43 under 35 U.S.C. 102(b) as being anticipated by Balogh et al. 
has been withdrawn as necessitated by amendment. 

4. Claims 22, 44 remain rejected under 35 U.S.C. 103(a) as being unpatentable over Balogh et al. 

5. Claims 1-9, 11-32, 34-44 are pending. Claims 10, 33 have been canceled by Applicant. Claims 1, 
16, 24, 38 are independent claims. 

Continued Examination Under 37 CFR 1.114 

6. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 
1 .17(e), was filed in this application after final rejection. Since this application is eligible for continued 
examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1. 17(e) has been timely paid, the 
finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's 
submission filed on 1 1/14/2002, and After Final amendment filed 10/28/2002, has been entered. 
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Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of 
this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as 
a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said 
subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made. 

8. Claims 1-9, 11-32, 34-44 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Balogh et ah (hereinafter Balogh), U.S. Patent No. 5,493,677 issued February 20, 1996. 

In regard to independent claim 1, Balogh teaches an application program comprising plurality 
of media clips in a database, with associated information describing each media clip, said media clips are 
in the form of images, as well as video clips and multimedia objects (Balogh Abstract, column 1 lines 56- 
64, column 3 lines 29-34, column 5 lines 48-57, Figure 6, 14; compare with claim 1 "A method for 
searching a media clip database associated with a multimedia application program, wherein said media 
clip database contains... . that describes each associated media clip in said media clip database, 
comprising:". 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image 
(Balogh column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar" - 
Balogh column 6 lines 38-44), and since Balogh teaches that keyword searching can be applied to 
querying (Balogh column 12 lines 8-14), as well as teaching that captions/bibliographic information can 
be reused for iterative querying (Balogh column 14 lines 40-60), said teachings provide a reasonable 
suggestion to one of ordinary skill in the art at the time of the invention, of the use of said metadata data 
as keywords, providing Balogh the benefit of querying with descriptive keywords for searching various 
image databases (compare with claim 1 "keywords"). 
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Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 
lines 1-7), resulting in retrieval of captions with images along with associated information (Balogh 
column 14 lines 3-10, column 16 lines 42-47, 56-67, Figure 12, 13) (compare with claim 1 "w response 
to a user selecting a media clip, retrieving information... . associated with said selected media clip from 
said media clip database"), 

Balogh teaches presenting the above captions, images and information to a user for eventual 
query (Balogh column 14 lines 3-10, 40-60, Figure 12-15; compare with claim 1 "presenting said 
keywords associated with said selected media clip for selection by the user "). 

Balogh teaches an additional query based upon the associated image description or bibliographic 
data of a previously selected image, the user can make further additions, deletions, and/or modifications 
to the associated information, if needed, prior to said additional query. It is to be noted that a user drags 
and drops a selected image into the description/bibliographic area, resulting in transfer of associated 
information (i.e. keywords, as explained above) to be used or edited in the additional query, resulting in 
retrieval of additional images (Balogh column 14 lines 40-59, column 16 lines 53-62; compare with claim 
1 "in response to the user selecting a search criteria by selecting one or more of said keywords 
associated with said selected media clip, retrieving all media clips in said media clip database that have 
associated keywords that match the one or more keywords associated with the selected media clip 
selected by the user"). 

In regard to dependent claims 2-4, Balogh teaches display of the best retrieved captions/ 
images displayed for user review. This process (including displaying a plurality of clip images) is 
repeated by said user, as desired (Balogh Figure 13-15, column 14 lines 3-10, 40-42, column 16 lines 53- 
62; compare with claims 2-4). 
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In regard to dependent claims 5-8, Balogh teaches a media type (images), as well as search 
criteria based upon color and/or shape (i.e. "red trucks", and "black cats" - keywords) (Balogh Abstract, 
column 1 1 lines 60-67; compare with claims 5-8). 

In regard to dependent claim 9, claim 9 reflects the combined subject matter of claims 7 and 8, 
and is rejected along the same rationale. 

In regard to dependent claims 11-12, Balogh teaches an additional query, whereby a result 
image is dragged and dropped into a description/bibliographic area, resulting in bibliographic (keyword) 
data copied to form a new query (in the case of Balogh, all of the data is selected) (Balogh column 14 
lines 40-59; compare with claim 11). 

Balogh teaches search criteria in the form of keywords as part of a search query, said query 
(keywords) can be based upon image characteristics or image type (Balogh column 1 1 lines 60-67, 
column 12 lines 9-14; compare with claim 12). 

In regard to dependent claims 13-14, Balogh teaches dragging and dropping of a selected 
image into a description/bibliographic area, resulting in a copy of the image's bibliographic data 
(keywords) into formulation of a new query (Balogh column 14 lines 49-59; compare with claim 13). 

Balogh teaches searching and presentation of files of type image from an image database (Balogh 
Abstract; compare with claim 14). 

In regard to dependent claim 15, a computer readable medium (i.e. diskette or hard drive) used 
for holding instructions is known in the art 
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In regard to independent claim 16, Balogh teaches an application program comprising plurality 
of media clips in a database, with associated information describing each media clip, said media clips are 
in the form of images, as well as video clips and multimedia objects. It is to be noted that Balogh also 
discloses associated data regarding what a particular media image suggests, which is indicative of a visual 
thesaurus (Balogh Abstract, column 1 lines 56-64, column 3 lines 29-34, column 5 lines 48-57, Figure 6, 
14, see also Figure 3 item 262, Figure 6 item 606, column 1 lines 59-61, column 3 lines 33-36; compare 
with claim 16 "A method for providing a user interface for a visual thesaurus for a media clip database 
associated with a multimedia application program, wherein said media clip database contains 
information that describes each associated media clip in said media clip database, comprising". 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image 
(Balogh column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar" - 
Balogh column 6 lines 38-44), and since Balogh teaches that keyword searching can be applied to 
querying (Balogh column 12 lines 8-14), as well as teaching that captions/bibliographic information can 
be reused for iterative querying (Balogh column 14 lines 40-60), said teachings provide a reasonable 
suggestion to one of ordinary skill in the art at the time of the invention, of the use of said metadata data 
as keywords, providing Balogh the benefit of querying with descriptive keywords for searching various 
image databases (compare with claim 16 "keywords"). 

Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 
lines 1-7), resulting in retrieval of captions with images along with associated information presenting said 
captions, images and information to a user (Balogh column 14 lines 3-10, 40-41, column 16 lines 42-47, 
56-67, Figure 12-15) (compare with claim 16 "directly in response to a user selecting a media clip from 
said media clip database"). 
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Balogh teaches an additional query based upon the associated image description or bibliographic 
data of a previously selected image, the user can make further additions, deletions, and/or modifications 
to the associated information, if needed, prior to said additional query. It is to be noted that a user has the 
option of visually dragging and dropping a selected image into the description/bibliographic area, 
resulting in transfer of associated information to be used or edited in the additional query, resulting in 
retrieval of additional images, said drag and drop is a visual indication of said option (Balogh column 14 
lines 40-59, column 16 lines 53-62; compare with claim 16 "displaying to the user an option for finding 
similar media clips that have an associated keyword that matches the associated keyword for the selected 
clip."). 

In regard to dependent claim 17, Balogh teaches a browsing tool for allowing a user to visually 
browse hits, as well as a number of "select" buttons for choosing certain candidate matches for further 
examination (Balogh column 16 lines 42-47, 55-60; compare with claim 17). 

In regard to dependent claim 18, Balogh teaches an additional query based upon the associated 
image description or bibliographic data of a previously selected image, the user can make further 
additions, deletions, and/or modifications to the associated information, if needed, prior to said additional 
query. It is to be noted that a user has the option of visually dragging and dropping a selected image into 
the description/bibliographic area, resulting in transfer of associated information to be used or edited in 
the additional query, resulting in retrieval of additional images, said drag and drop is a visual indication of 
said option (Balogh column 14 lines 40-59, column 16 lines 53-62; compare with claim 18). 

In regard to dependent claims 19-21, 23, a fly-out window (i.e. an overlaying window, or 
balloon help annotation with additional information, etc.), is known in the art (compare with claim 19). 
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An option for inserting an image into a document (i.e. clipboard copy and paste), is known in the 
document processing art (compare with claim 20). 

Balogh teaches a browser for vewing image hits, said hits comprise a thumbnail (preview) image 
along with a caption "snippet" from each image (Balogh column 16 lines 63-67, column 17 lines 1-13; 
compare with claim 21). 

A computer readable medium (i.e. diskette or hard drive) used for holding instructions is known 
in the art (compare with claim 23). 

In regard to dependent claim 22, Balogh does not specifically teach an option to add a clip to a 
category. However, this limitation would have been obvious to one of ordinary skill in the art at the time 
of the invention, in view of Balogh, because Balogh teaches ordering selected images through a 
purchase/delivery service (Balogh column 17 lines 40-44). In the case of Balogh, images are user selected 
for purchase (a user voluntarily adding a media clip to a category intended for purchase) (see also Balogh 
column 17 lines 48-50, and column 18 lines 1-10). Adapting Balogh to incorporate user inclusion of clips 
into various additional categories as taught by Balogh, provides a user of Balogh the benefit of grouping 
selected images for further action (i.e. negotiation, reservation, trade, etc.). 

In regard to independent claim 24, Balogh teaches a data entry, disambiguation, and database 
processors within a SUN SPARCSTATION (Balogh column 3 lines 60-67; compare with claim 24 "a 
processing unit"). 

Balogh teaches a computer readable medium (i.e. diskette or hard drive) used for holding 
instructions and utilized within a computer, is known in the art (compare with claim 24 "a storage 
medium... .by the processing for"). 

Balogh teaches an application program comprising plurality of media clips in a database, with 
associated information describing each media clip, said media clips are in the form of images, as well as 
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video clips and multimedia objects, said clips subject to user query and selection (Balogh Abstract, 
column 1 lines 56-64, column 3 lines 29-34, column 5 lines 48-57, column 16 lines 56-67, column 17 
lines 1-13, Figure 6, 14; compare with claim 24 "providing an interface for a user to select a media 
clip... .in said media clip database"). 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image 
(Balogh column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar 55 - 
Balogh column 6 lines 38-44), and since Balogh teaches that keyword searching can be applied to 
querying (Balogh column 12 lines 8-14), as well as teaching that captions/bibliographic information can 
be reused for iterative querying (Balogh column 14 lines 40-60), said teachings provide a reasonable 
suggestion to one of ordinary skill in the art at the time of the invention, of the use of said metadata data 
as keywords, providing Balogh the benefit of querying with descriptive keywords for searching various 
image databases (compare with claim 24 "keywords"). 

Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 
lines 1-7), resulting in retrieval of captions with images along with associated information, as well as an 
additional query based upon the associated image description or bibliographic data of a previously 
selected image, the user can make further additions, deletions, and/or modifications to the associated 
information, if needed, prior to said additional query. It is to be noted that a user drags and drops a 
selected image into the description/bibliographic area, resulting in transfer of associated information to be 
used or edited in the additional query, resulting in retrieval of additional images (Balogh column 14 lines 
40-59, column 16 lines 42-47, 53-67, Figures 12-13; compare with claim 24 providing an interface for the 
user to select... .with said selected media clip", and "in response to the user... .for the selected media 
clip."). 
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In regard to dependent claims 25-32, 34-37, claims 25-32, 34-37 reflect the apparatus 
comprising computer readable instructions used for performing the methods as claimed in claims 2-9, 1 1- 
14, respectively, and are rejected along the same rationale. 

In regard to independent claim 38, Balogh teaches an application program comprising plurality 
of media clips in a database, with associated information describing each media clip, said media clips are 
in the form of images, as well as video clips and multimedia objects. It is to be noted that Balogh also 
discloses associated data regarding what a particular media image suggests, which is indicative of a visual 
thesaurus (Balogh Abstract, column 1 lines 56-64, column 3 lines 29-34, column 5 lines 48-57, Figure 6, 
14, see also Figure 3 item 262, Figure 6 item 606, column 1 lines 59-61, column 3 lines 33-36; compare 
with claim 38 "An apparatus for providing a user interface for a visual thesaurus for a media clip 
database associated with a multimedia application program, wherein said media clip database contains 
information... . that describes each associated media clip in said media clip database, comprising", 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image 
(Balogh column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar" - 
Balogh column 6 lines 38-44), and since Balogh teaches that keyword searching can be applied to 
querying (Balogh column 12 lines 8-14), as well as teaching that captions/bibliographic information can 
be reused for iterative querying (Balogh column 14 lines 40-60), said teachings provide a reasonable 
suggestion to one of ordinary skill in the art at the time of the invention, of the use of said metadata data 
as keywords, providing Balogh the benefit of querying with descriptive keywords for searching various 
image databases (compare with claim 38 "keywords"). 

Balogh teaches a data entry, disambiguation, and database processors within a SUN 
SPARCSTATION (Balogh column 3 lines 60-67; compare with claim 38 "a processing unit"). 
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Balogh teaches a computer readable medium (i.e. diskette or hard drive) used for holding 
instructions and utilized within a computer, is known in the art (compare with claim 38 "a storage 
medium... .by the processing unit for... "). 

Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 
lines 1-7), resulting in retrieval of captions with images along with associated information presenting said 
captions, images and information to a user, as well as an additional query based upon the associated 
image description or bibliographic data of a previously selected image, the user can make further 
additions, deletions, and/or modifications to the associated information, if needed, prior to said additional 
query. It is to be noted that a user has the option of visually dragging and dropping a selected image into 
the description/bibliographic area, resulting in transfer of associated information to be used or edited in 
the additional query, resulting in retrieval of additional images, said drag and drop is a visual indication of 
said option (Balogh column 14 lines 3-10, 40-59, column 16 lines 42-47, 53-67; compare with claim 38 
"... displaying to the user an option for finding similar media clips that have associated keywords that 
matches the associated keywords for a selected media clip, directly in response to the user selecting the 
media clip'"). 

In regard to dependent claims 39-43, claims 39-43 reflect the apparatus comprising computer 
readable instructions used for performing the methods as claimed in claims 17-21, respectively, and are 
rejected along the same rationale. 

In regard to dependent claim 44, claim 44 reflects the apparatus comprising computer readable 
instructions used for performing the methods as claimed in claim 22, and is rejected along the same 
rationale. 
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9. 



Prior art made of record and not relied upon is considered pertinent to disclosure. 



Hoffert et al. 
De La Huerga 



U.S. Patent No. 6,282,549 
U.S. Patent No. 6,308,171 



issued 
issued 



08-2001 
10-2001 



Response to Arguments 



10. Applicant's arguments with respect to claims 1-44 have been carefully considered but are not 
persuasive. 

Applicant argues on page 4 of the amendment regarding an apparent error in the examiner's 
heading of 35 U.S. C. 102(b). This error is respectfully acknowledged, but is now moot in view of the new 
grounds of rejection. 

Applicant argues on page 5 (and is repeated on pages 8-19) of the amendment that Balogh fails to 
teach keywords. The examiner respectfully disagrees. Balogh's "captioner" provides each image with 
metadata in the form of a caption describing salient features of an image, bibliographic data, "suggest 
fields" and attributes, for each image Since said metadata can include descriptive words, as well as 
teaching that captions/bibliographic information can be reused for iterative querying. Since Balogh 
teaches that keyword searching can be applied to querying (Balogh column 12 lines 8-14) said teachings 
provide a reasonable suggestion to one of ordinary skill in the art at the time of the invention, of the use of 
said metadata data as keywords. In addition, the metadata is displayed to a user for subsequent querying 
of images. Although Balogh teaches an embodiment comprising natural language sentence captions, the 
words of said sentences are suggestive of keywords, since Balogh teaches parsing a query into individual 
tokens representing single words or multi words, said words subsequently matched to various captions (as 
well as other metadata) within the search process (see Balogh column 12 lines 33-37). 



Application/Control Number: 09/224,009 Page 13 

Art Unit: 2176 

Applicant's argument regarding the rejection of claims 6, 29 under 35 U.S.C. 1 12 Second 
Paragraph is currently moot as necessitated by amendment. 

Applicant argues on page 7 of the amendment that Balogh does not teach retrieving information 
in response to a user selecting a media clip. The examiner respectfully disagrees. Balogh teaches a user 
can drag and drop a selected image (or a media clip) into a description area, the information (caption 
and/or bibliographic data associated with said image) is then copied into a new query for further search 
(Balogh column 14 lines 40-60). 

Applicant argues on page 10 of the amendment that Balogh does not teach search criteria as 
"shapes". The examiner respectfully disagrees. Balogh teaches an example search criteria: "a black caf\ 
which can be interpreted as finding images with the color and shape of a black cat, because cats have a 
basic inherent shape. 

Applicant argues on page 1 1 of the amendment that Balogh does not teach an indication of 
options displayed to a user for further search. The examiner respectfully disagrees. Since dragging and 
dropping an image for further query is part of Balogh's disclosed invention, a user has an option to 
manually and visibly invoke further search. 

Conclusion 

11. Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to William Bashore whose telephone number is (703) 308-5807. The examiner can 
normally be reached on Monday through Friday from 1 1:30 AM to 8:00 PM EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Heather Herndon, can be reached on (703) 308-5186. 

Any inquiry of a general nature or relating to the status of this application should be directed to 
the Group receptionist whose telephone number is (703) 305-3900. 
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12. Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 

or faxed to: 

(703) 746-7239 (for formal communications intended for entry) 

or: 

(703) 746-7240 (for informal or draft communications, please label 
"PROPOSED" or "DRAFT") 



(703) 746-7238 (for after-final communications) 

Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal Drive, 
Arlington, VA, Fourth Floor (Receptionist). 



or: 



William L. Bashore 
01/25/2003 




